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1.  INTRODUCTION 

The  last  decade  has  seen  the  growth  of  videspread  computer  use  in  a 
large  variety  of  applications  in  may  different  fields  of  specialisation.  As 
a  result,  the  suppliers  of  computing  services  have  been  plunged  more  and  more 
deeply  into  the  task  of  providing  small  amounts  of  computing  time  for  large 
numbers  of  users.  In  general,  two  solutions  are  possible:  provide  a  small 
computer  for  each  individual  user  or  local  group  of  users,  or  a  few  large 
computers  providing  service  wholesale  to  many  users  at  many  remote  terminals. 
Compromises  between  the  two  are  possible,  and  sometimes  even  desirable. 

Examples  are  large  secondary  memories  shared  among  many  small  computers,  and 
remote  terminals  of  a  large  computer  with  computing  capabilities  of  their  own. 

Information  systems  (i.e.,  "mechanized  libraries")  designed  to  make  large 
files  of  Information  easily  accessible  to  may  users,  most  of  whom  require  infre¬ 
quent  access,  are  forced  into  the  category  of  large -processor  terminal  systems. 
The  teletypewriter  (TTY)  is  currently  the  .east  expensive  and  most  widely  distri¬ 
buted  remote  terminal.  Commercial  communication  lines  provide  relatively 
inexpensive  remote  communication  for  infrequently  used  terminals.  However,  the 
TTY-comoercial  lies  combination  presents  a  number  of  problems  to  the  designers 
of  information  systems.  Some  of  these  are  discussed  in  this  paper.  Most 
particular  attention  is  paid  to  the  interactive  (man-machine)  language  for 
TTY,  and  some  of  the  mechanical  and  psychological  limitations  in  their  design. 
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The  primary  search  activities  of  *  library  user  can  be  subdivided  into 
three  distinct  categories:  first,  he  nay  desire  to  ask  for  an  explanation  of 
the  indexing  or  classification  system  used,  or  for  a  guide  to  the  index  terns 
or  classification  subject  headings,  or  for  some  portion  of  these  vhich  holds 
his  particular  interest;  second,  he  nay  search  the  catalog  (or  index)  eyetem 
for  accession  numbers  of  documents  which  are  likely  to  fulfill  his  needs;  and 
third,  he  may  withdraw  documents  thus  pinpointed  from  the  shelves  for  browsing 
or  decision  as  to  actual  relevance  to  the  problem  at  hand. 

These  three  search  activities  may  be  repeated  any  number  of  tinea  in  some 
arbitrary  sequence  until  the  search  has  been  completed.  Depending  on  the 
thoroughness  of  the  search,  the  searcher  reaches  a  certain  level  of  confidence 
that  he  has  a  list  of  accession  numbers,  and  possibly  a  set  of  documents, 
vhich  will  (best)  fulfill  his  information  requirements. 

The  current  research  project  has  engaged  in  some  experiments  on  Informa¬ 
tion  retrieval  from  a  computerized  file  where  communication  was  established 
through  an  on-line  typewriter.  An  information  retrieval  system  which  provides 
for  uan -machine  dialogue  at  a  remote  inquiry  terminal  should  provide  a  searcher 
with  many  of  the  tools  which  would  be  available  to  him  were  he  actually 
performing  this  search  at  a  library  or  repository  of  documents.  He  should  have 
the  capability  of  tracking  down  documents  of  interest  to  him  in  much  the  same 
way  he  would  do  Hln  person11  and  probably  with  greater  ease.  To  this  end,  the 
system  should  provide  him  with  the  same  three  primary  search  tools: 

(l)  Library  structure:  information  about  the  organisation  of  the 
document  file  and  access  to  any  indexing  or  classification  tools 
eaqployed; 
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(2)  Document  retrieval:  ability  to  partition  the  document  file  into 
•etc  of  documents  of  varying  degrees  of  relevance  to  his  search, 
as  a  function  of  the  information  provided  by  the  searcher;  and 

(5)  Document  perusal:  ability  to  "peruse"  documents  which  Bight  be  of 
interest,  or  at  least  to  request  certain  information  about  eoch 
documents,  such  as  bibliographic  data  or  abstracts. 

A  mechanized  library  should  provide  a  remote  user  information  about  ite 
structure  ae  well  as  instruction  in  its  use.  Most  important,  the  scheme 
employed  in  retrieval  of  documents,  or  in  partitioning  the  document  file  into 
document  sets  of  varying  relevance  with  respect  to  a  particular  request,  should 
be  readily  available  to  the  user.  If  a  classification  scheme  ie  employed,  the 
"classification  tree"  should  be  available  for  inspection  by  the  user  upon 
request  and  in  an  informative  and  easily  digestible  form.  In  a  coordinate 
indexing  environment  with  controlled  vocabulary,  the  user  should  be  provided, 
if  possible,  with  means  to  inspect  the  thesaurus,  together  with  any  semantic 
tools  used  in  indexing.  Furthermore,  instruction  in  the  use  of  the  system  and 
its  command  language  should  be  available  in  some  form.  If  possible,  the  system 
should  “bootstrap  the  user  in,"  or  assist  a  neophyte  user  who  has  no  previous 
experience,  much  the  same  as  the  librarian  instructs  a  new  user  to  the  tradi¬ 
tional  library. 

These  characteristics  become  not  only  necessary,  but  vital,  in  an  open 
system,  where  any  teletypewriter  with  data-phoce  connection  may  be  used  as  a 
remote  terminal.  At  seme  time  in  the  future,  such  an  information  system  and 
its  command  language  my  be  widespread,  and  its  rules  of  operation  will  be 
common  knowledge.  Until  such  time,  the  system  had  better  be  simplicity  itself 
if  it  is  in  any  way  to  compete  with  standard  library  research  technique 
[1,5,7,9,13]. 
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Clarity  1b  important  because  results  should  be  understood  the  first  time 
they  are  printed.  Printout  takes  long  enough  as  is,  and  although  concise,  the 
format  should  not  leave  room  for  misunderstanding.  Codes  or  abbreviations  used 
should  be  carefully  defined,  and  avoided  where  ever  possible.  The  balance 
between  clarity  and  conciseness  is  not  always  apparent. 

2 .  MCIMEHT  PERUSAL 

The  traditional  library  provides  the  user  with  the  ability  to  peruse 
documents  located  (e.g.,  removed  from  the  shelf)  as  &  re cult  of  a  search. 
Further,  card  catalogue  cards  may  be  perused  for  subject  index  and  bibliographic 
data  and  sometimes  short  descriptions.  Reviews,  summaries,  and  abstracts  may 
be  perused  in  appropriate  journals.  These  procedures  assure  that  material 
checked  out  or  ordered  from  the  library  is  relevant  and  of  interest.  A  remote 
terminal  system  must  provide  similar  facilities.  Although  perusal  of  documents 
themselves  is  somewhat  beyond  the  scope  of  the  present  state  of  the  art,  compact 
descriptive  material  for  each  document  must  be  available  for  perusal.  This 
information  must  necessarily  require  only  small  amounts  of  storage  over  and 
above  that  already  required  for  retrieval.  A  duality  exists  here--,  information 
characterizing  a  document  for  retrieval  characterizes  it  for  perusal  as  well. 
Bibliographic  data,  index  terms,  and  abstracts  are  all  useful  In  both  retrieval 
and  perusal.  The  difficulty  lies  in  file  structure  design  and  formatting  so 
that  the  desired  information  can  be  requested  in  a  reasonable  way  and  prated  in 
a  reasonable  format  within  reasonable  time  limitations. 

A  remote  information: system  operating  in  conversational  mode  with 
document  perusal  available  in  real  time  allows  the  user  to  "zero  in"  on  the 
documents  of  interest  to  him,  and  to  choose  with  a  high  degree  of  confidence. 

The  actual  retrieval  algorithms  or  procedures  are  of  less  Importance,  since  the 
retrieval  procedure  is  interactive  and  iterative. 
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b.  CCWAKD  IAKGUAOE 

A  mechanized  information  retrieval  system  must  employ  &  command  language. 
Such  m  language  cftn  be  classified  According  to  itB  orientation,  i.e.,  system- 
oriented  language  would  presumably  employ  many  curt,  arbitrary  symbols  to  repre¬ 
sent  various  operations,  and  a  syntax  closely  akin  to  the  logical  structure  of 
the  mechanized  system*  A  language  which  is  more  user-oriented  would  be  provided 
with  syntax  or  construction  of  statements  and  requests  more  akin  to  the  logical 
formulation  of  requests  by  people.  Certainly,  the  most  user-oriented  language 
possible  is  natural  English,  or  any  other  natural  language  that  a  searcher 
would  ordinarily  employ  for  communication  with  a  librarian.  Of  course,  there 
are  varying  degrees  of  user  and  system  orientation. 

Command  languages  can  also  be  measured  according  to  their  flexibility  cr 
general  applicability.  A  large  variety  of  test  or  pilot  information  systems 
employ  different  specialized  capabilities.  A  system  that  can  be  thought  of  as 
having  some  sort  of  general  applicability  should  easily  accommodate  differing 
points  of  view  or  system  capabilities,  such  as  binary  or  multiple  relevance 
rating  capabilities,  and  a  large  set  of  retrieval,  specification  parameters 
(bibliographic  specification,  key  word  in  context,  subject  index,  coordinate 
index,  etc.). 

Naturally,  there  is  a  tradeoff  of  time,  load  capability  and/or  efficiency 
with  respect  to  the  d  .  of  flexibility  and  user  orientation  of  a  command 
language.  A  language  which  allows  statements  in  a  limited  subset  of  English 
is  necessarily  slow  and  requires  laige  scale  mechanization  and  a  large  storage 
Just  for  passing  and  translating  statements.  A  highly  efficient,  limited  capa¬ 
bility  language  requires  greater  learning  time  on  part  of  the  user,  and  there¬ 
fore  night  risk  a  smaller  subscribership,  Some  balance  is  necessary  in  the 
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choice  of  a  language  commensurate  with  its  intended  use  and  yet  consistent  with 
available  computation  and  storage  devices.  This  is  a  particularly  important 
question  in  a  system  designed  for  thousands  of  users,  mane  of  whom  need  access 
very  1 nf re quant ly  (maybe  only  once  or  twice  annually).  A  language  that  is 
difficult  to  learn  would  drastically  repel  users  who  require  only  occasional 
access. 

U.  THE  MOORE  SCHOOL  EXPERIMENTAL  INFORMATION  SYSTEM 

An  information  system  responsive  to  many  of  the  preceding  guideline# 
has  been  set  up  by  the  Moore  School  Information  System  Laboratory  (MSISL), 
University  of  Pennsylvania.  The  document  set  consists  of  the  Repository  of 
the  Association  of  Computing  Machinery  (about  1500  documents).  The  interac¬ 
tive  language  includes  a  section  of  computer-directed  search  with  computer 
control  exercised  through  a  sequence  of  "yes-no"  type  questions  and  a  message 
insertion  section  with  a  special  request  language  ("Symbolic  Language")  employed 
by  the  searcher  in  specifying  a  document  retrieval  request.  Three  types  of 
requests  are  available,  termed  "Retrieve",  "Combine",  and  "Display".  "Retrieve" 
allows  the  searcher  to  specify  a  retrieval  request  by  a  logical  combination  of 
descriptors;  "Combine"  elicits  retrieval  of  a  set  of  documents  characterized  by 
one  or  more  of  a  set  of  descriptors;  and  "Display"  causes  retrieval  of  a  set  of 
documents  specified  by  accession  number. 

It  is  sufficient  for  present  purposes  to  refer  to  Fig.  1  for  a  syntactic 
description  of  Symbolic  Language,  Figs.  2a-2d  for  an  illustration  of  an  actual 
on-line  search,  and  Fig.  5  for  a  semantic  description  of  various  elements  of  a 
retrieval  request.  In  Figs.  2a  and  2b  a  request  is  made  for  documents  charac¬ 
terized  by  one  or  more  of  the  five  items  IBM,  PROGRAMMING,  MANUAL,  FORTRAN,  and 
FEB  1955.  The  first  four  are  supposedly  index  terms  and  the  fifth  is  a 
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Figure  1 

Backus  normal  Specification  of  Symbolic  language  Syntax 


<dffi(g£83> !  :»<W5tRlEV13  R£S>  <£HD>  {  <CCWBINE  RE«>  <£NE>  |  DISPLAY  fCCG>  <ENI3> 
4iE1KXfUt  RSTftlB VE  <R  SPEC3> 

<R  SPECS>:  >  <CATEGORY>  <TERK>  |  <K  8F£CS>  <0FERAT0R>  <R  SPECS> 
<0PESAT0R>: :«  +|ijl 

<CATEOORY>: $AO|$Al|$A2|$A3|$A4|$A5}$A6t$A7|$A8|$A9|$B|$C 
<TERK>:  :•  <3TR3KO>  j  (<TEH Ha  <OFERATOK>  <TEWt>) 

<STRINO: >  <VOKD>  |  <STRDK>  <WORD> 

<WOR]J>::»i  Any  set  of  up  to  24  contiguous  characters 
*rott  the  alphabet)  the  decimal  digits, 
hyphen,  asterisk,  period,  comma,  apostrophe, 

<CCMBXHE  BEQ>: COMB  DIE  <C  SJECS> 

<C  SPECS>:  :•  <CATEGOBY>  <C  SEQUENCER  |  <C  SFECS>  /  <C  SFECS> 

<C  SEQUENCE>:  :«•  <STRHKJ>  |  <C  SEQUENCE*  /  cSTBDKt* 

<DIBPLAY  HE<J>: DISPLAY  (<ACCESSIOM  UST>) 

<ACCE8SI0N  LIST>: <ACC  NTJKBER>  \  ACCESSION  LIST>,  <ACC  HUKBEEO 
<ACC  HIKBEH>::«  <NUMBEF>  j  <NUMBEK>  -  <EI0IT> 

<DIGn>:  :■  One  of  the  ten  decimal  digits, 

<HUHBEI&> : j»  An  integer,  from  one  to  four  digits, 

4DCD>: :•  The  sequence  of  two  teletypewriter  symbols  "<  >". 

Bote:  The  elements  of  a  Symbolic  language  statement  as  specified  above 
should  be  separated  by  one  or  more  blanks.  Carriage  returns  are 
considered  blanks,  so  that  elements  may  not  be  continued  from  one 
line  to  another.  The  one  exception  to  this  rule  la  in  writing 
accession  numbers  with  appended  digits.  In  this  case,  there  are 
no  blanks  inserted,  as  for  example  --  953-7.  Furthermore,  a 
character  'V'  immediately  following  it;  a  line  of  characters  My 
be  erased  by  typing  V,'"{  an  entire  message  may  be  erased  by 
typing  V?". 
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Condensed  Semple  of  Classification  Table  for  Synonyms 
(from  table  approximately  alx  tinea  Chi*  size) 

*igure  2a. 


Condensed  Sample*  of  Clarification  Table* 
Figure  2b 
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Coodanacd  Staple  of  Claaalftcaeioii  Table 
Figure  2d 


12  " 


Figure  3 

Semantic  description  of  Symbolic  language 


A  string  (<STRU¥G>)  is  a  string  of  words  comprising  the  descriptor  of 
one  or  more  documents.  A  substring  of  such  a  descriptor  is  also  accept* 
able  as  a  descriptor,  e.g. ,  part  of  a  title  or  one  word,  of  a  two  word 
term. 

Accession  numbers  (<ACC  HUHBER>)  are  unique  numbers  in  tbs  format 
defined  in  Fig.  1  assigned  to  each  document  in  the  collection. 

The  worda  "RETRIEVE",  "COMBINE",  and  "DISPIAY"  Indicate  the  type  of 
retrieval  desired. 

The  following  action  codes  indicate  categories  under  which  documents  are 
Indexed  (<CA'ffiOORY>) : 


0  -  Pate  of  indexing  and  indexer's  code  iA5  • 

1  •  Authors  $A6  - 

2  -  Date  of  Issue  iA7  - 

3  -  Title  sA8  - 

U  -  Editors,  etc.  $A9  - 

•  Index  terms 

-  Added  information  (Special  codes  provide 
a  large  variety  of  additional  information 
about  documents.  Code  list  is  available). 


g  : 


3  -  Issuers 

6  -  Page  site 

7  -  Humber  of  illustrations 

8  -  Humber  of  pages 

9  -  Collection  In  which 

article  appears 


The  following  infix  binary  connectives  are  used  to  combine  terms  (<Operator>) 

A  AND  (logical  and,  A) 

♦  OR  (logical  exclusive  or  V) 

t  AND  NOT  (logical  co-lmpllcaTTon,  or  inhibition,  J») 

A  list  of  accession  numbers  of  documents  is  actually  produced  as  a  result 
of  a  retrieval  request.  The  searcher  is  then  queried  as  to  what  (if 
anythin)  he  would  like  to  peruse  (have  printed  out)  concerning  these 
documents  on  the  list 
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publication  date.  After  the  retrieval*  the  accession  numbers  of  those  charac¬ 
terized  by  at  least  three  of  these  five  items  is  requested  and  printed*  Figure  3c 
illustrates  the  retrieval  of  four  documents  authored  by  either  PATTERSON  or 
CARR*  J.V,  and  the  printout  of  all  bibliographic  information  available  concerning 
these  documents.  Figure  3d  the  authors  (where  available)  and  titles  of  four 
documents  listed  by  accession  number  are  printed. 

Some  of  the  important  features  of  the  system  are  as  follows: 

1.  Response  time  of  the  system  to  most  requests  (unless  they  are 
exceedingly  complex*  and  unlikely  in  a  real  library  search)  is  on 
the  order  of  one  to  five  seconds* 

2.  Messages  may  be  edited  while  typing  (see  note  on  Fig.  l).  They  may 
also  be  edited  after  the  message  is  completed,  if .desired, 

3*  The  descriptors  used  as  characterization  of  a  document  for  retrieval 
purposes  are  available  for  perusal  of  retrieved  documents* 

b.  Retrieval  and  perusal  may  be  alternated  any  number  of  times  in 
deciding  upon  exactly  the  desired  documents* 

The  experimental  system  is  not  time-shared*  but  is  accessible  to  any 
teletypewriter  with  a  data-phone  connection.  Tests  have  been  run  from  as  far  as 
Boulder*  Colorado,  by  standard  telecommunication  lines.  (The  system  is  of  course 
located  in  Philadelphia  at  the  University  of  Pennsylvania.) 

3.  SOME  QUESTIONS 

Some  subjective  as  well  as  objective  questions  arose  with  regard  to  the 
MSISL  system*  In  such  a  system  easy  to  use?  Does  a  searcher  feel  comfortable 
using  it?  How  quickly  can  a  person  with  no  background  in  information  retrieval 
learn  to  use  the  system?  Vhat  form  of  Instruction  in  itB  use  is  most  fruitful? 
How  effective  is  the  retrieval  language  for  efficient  and  relevant  retrieval? 


Where  does  it  fail  to  emulate  the  traditional  library  search? 

Noting  that  such  answers  with  respect  to  a  small  pilot  system  do  not 
necessarily  provide  the  answers  with  respect  to  a  large  multi-user  system  with 
a  large  document  collection,  but  might  nonetheless  provide  some  dues  as  to 
larger  system  design,  several  experiments  were  performed.  These  experiments 
did  in  fact  provide  some  guidelines  for  future  design  of  interactive  systems 
of  this  sort. 

6.  THE  EXPERIMENTS 

The  experiments  were  conducted: 

I.  The  purpose  of  this  test  was  to  study  the  ease  of  familiarization 
with  the  capability  of  the  system,  the  ease  cf  developing  working 
retrieval  strategy,  and  to  reveal  the  drawbacks  of  the  system  from 
the  psychological  viewpoint  of  the  user. 

Subjects  were  given  a  retrieval  task,  told  of  the  capability 
of  the  retrieval  system,  but  not  taught  the  command  language.  They 
verbally  transmitted  requests  to  a  teletype  operator  who  translated 
these  to  the  command  language.  The  subjects  saw  the  system  output  in 
this  way  formulated  further  requests.  Their  gradual  familiarization 
with  the  system  was  then  studied  by  ar.  examination  of  the  development 
of  their  search  strategy. 

Appendix  A  presents  the  instructions  given  to  the  subjects 
for  the  purposes  of  this  test. 

II.  The  purpose  of  this  test  was  to  study  the  ease  or  difficulty  of 
teaching  the  request  (command)  language.  Appendix  2  presents  the 
instruction  sheet  given  subjects  in  this  test. 
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Subjects  were  given  a  written  booklet  of  instructions  for  use 
of  the  system,  including  the  syntax  and  semantics  for  the  command 
language.  The  programmed  instruction  routine  was  still  available  at 
the  typewriter  keyboard. 

III.  The  purpose  of  this  test  was  similar  to  that  of  test  II,  but  was 

also  intended  to  reveal  whether  complete  instructions  for  the  use  of 
the  system  could  be  assimilated  or  partially  assimilated  before 
actual  trial. 

Subjects  were  given  a  retrieval  task  and  told  nothing  about  the 
system  except  for  the  initial  identification  code  and  a  general 
description  of  its  capabilities.  A  programmed  instruction  routine 
was  included  in  the  system  which,  sensing  no  acceptable  input 
message  after  initial  identification,  inquired  as  to  whether  the 
operator  desired  instruction  in  the  use  of  the  system.  Ar.  affirma¬ 
tive  reply  elicited  the  teaching  routine. 

In  addition  to  a  study  of  the  actual  interactive  dialogue,  each  of  the 
subjects  was  interviewed  subsequent  to  the  experiment. 

7.  CONCLUSIONS  AID  FUTURE  PLANS 

The  combined  results  of  the  three  experiments  can  be  summarized  as 
follows : 

a)  A  minimum  of  30  minutes  is  required  to  learn  even  the  rudiments  of 
the  command  language  (Symbolic  Language).  This  did  not  vary  sub¬ 
stantially  between  graduate  students  in  the  sciences  and  securities, 
between  unose  familiar  with  programming  languages  and  those  not. 
Often,  a  good  deal  of  time  is  necessary  for  familiarization  with  the 


teletype  keyboard 
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b)  As  in  most  other  tasks,  although  written  instructions  are  helpful, 
practice  is  essential, 

c)  The  programmed  instruction  routines,  which  took  approximately  20 
minutes,  were  mostly'  of  the  "yes-no"  variety.  Expanded  routines, 
which  would  require  other  replies  and  would  give  student-users 
actual  practice  in  formation  of  retrieval  requests  would  have  been 
better,  but  would  have  taker,  considerably  more  time  (and  would  have 
tried  the  patience  of  more  astute  students), 

d)  Flexibility  was  sacrificed  for  the  sake  of  simplicity  in  the  command 
language,  and  this  was  pointed  out  as  a  definite  deficiency  by  many 
of  the  subjects.  Increased  flexibility  would  have,  of  course, 
added  to  the  required  learning  time. 

e)  The  slow  typing  speed  of  the  teletypewriter  (slower  than  the  average 
reading  rate)  and  the  high  noise  li-vel  were  pointed  out  as  distrac¬ 
ting  factors  by  many  of  the  subjects. 

Some  of  the  difficulties  encountered  can  certainly  be  eliminated  by 
redesign.  Careful  choice  of  messages  can  assure  both  conciseness  and  clarity. 
Back-up  messages  can  explain  in  greater  detail  what  the  concise  messages  leave 
unclear  to  any  user.  Provisions  may  be  made  for  termination  of  long  printout 
in  "Midstream"  when  the  user  discovers  he  is  on  the  wrong  track  or  in  error. 

Certain  difficulties,  however,  are  inherent  in  the  limitations  of  the 
teletypewriter  as  a  remote  console.  Heading  rates  are  much  greater  than  the 
printout  rates,  particularly  in  long  printouts.  Often  the  contents  of  much  of 
the  printout  is  guessed  before  it  actually  appears  on  the  paper.  For  instance, 
when  inquiring  for  the  publisher  of  a  book,  the  letters  "MCGR"  often  suffice  to 
convey  "MCGRAW  HILL  BOOK  COMPANY",  and  the  remaining  two  seconds  of  printout 
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are  not  only  wasted,  but  are  annoying  to  the  user.  Furthermore,  the  noise 
level  of  the  telet/p evriter  is  high  and  promotes  anything  but  clear  thinking 
and  concentration.  Backspacing  and  erasures  are  not  possible,  and  the  V" 
editing  feature  does  not  entirely  sake  up  for  this  deficiency.  These  are  pro¬ 
blems  ve  intend  currently  to  live  with,  hoping  that  the  advantages  of  remote 
information  consoles  offset  the  disadvantages. 

The  command  language  is  more  readily  adaptable.  It  la  planned  to  replace 
Symbolic  Language,  employed  in  this  experiment,  by  a  command  language  called 
"Real  English,"  which  employs  real  English  sentences  in  a  specified  format. 

Pilot  tests  show  that  such  a  request  language  is  powerful  enough  to  handle  a 
majority  of  requests,  and  is  easily  mastered  by  neophytes. 

Without  entering  into  the  syntactic-semantic  structure  of  the  "Real 
English"  developed  to  date,  we  shall  present  a  few  examples  of  a  request  state¬ 
ment  which  are  acceptable  statements  in  the  language.  All  five  statements  below 
will  cause  the  same  retrieval,  and  are  considered  identical  by  the  system 
after  passing. 

PLEASE  LOCATE  EVERYTHING  WRITTEN  BY  ROBERT  PERKINS  ABOUT  EASIAC  OR 
PSEUDO-COMPUTERS  BETWEEN  1955  and  1859  <  > 

COULD  YOU  FIND  FOR  ME  SOMETHING  0,'SITAXNSD  TJ  THE  REPOSITORY  CONCERNING 
EASIAC  OR  PSEUDO-COMPUTERS  THAT  WAS  AUTHORED  BY  ROBERT  PERKINS  AFTER 
1951*  AND  BEFORE  i960  c  > 

I  NEED  ALL  THE  AVAILABIE  DOCUMENTS  PUBLISHED  DURING  THE  PERIOD  1955 
TO  1959  BY  ROBERT  PERKINS  ON  THE  SUBJECTS  OF  EASIAC  OR  PSEUDO-COMPUTERS  <  > 

WE'RE  INTERESTED  IN  HAVIN'!  REFERENCES  AND  MATERIAL  ON  EITHER  PSEUDO¬ 
COMPUTERS  OR  EASIAC  AUTHORED  BY  ROBERT  PERKINS  FROM  1955  TO  1959  <  > 

I  WOULD  LIKE  YOU  TO  HELP  ME  OBTAIN  INFORMATION  FRCM  YOUR  LIBRARY  RELATED 
TO  EASIAC  OR  PSEUDO-COMPUTERS  AND  WRITTEN  BY  ROBERT  PERKINS  IN  THE  YEARS 
1955  THROUGH  1959  <  > 

Instruction  routines  in  the  use  of  this  command  language  will  be  built  into  the 
system  and  further  tests  will  be  run  to  check  their  effectiveness.  Unacceptable 


statements  will  cause  the  system  to  query  the  user  as  to  his  intentions,  ask 
for  synonyms  of  words  not  included  in  the  system's  semantic  dictionary,  and 
request  the  user  to  restate  his  question  in  different  words.  Hopefully,  this 
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APPEND  DC  A 

Directions  for  Test  Study  I 

THE  PURPOSE:  We  are  testing  the  effectiveness  of  the  present  information- 
retrleval  apparatus.  You  are  helping  us  test  the  system  performance  in  a 
simulated  retrieval  situation. 

THE  TASK:  We  vould  like  you  to  compile  &  list  of  documents  vhich  will  be 
helpful  in  writing  a  paper  on  "Artificial  Languages."  The  paper  will  discuss 
numerous  languages  and  will  mention  such  items  as  algebraic  languages,  list 
processing,  and  business-oriented  languages.  The  paper  will  emphasize  the 
history  of  the  development  of  automatic  programming,  You  should  use  the 
retrieval  system  to  collect  the  information. 

THE  METHOD:  The  following  explanation  describes  the  system,  and  it  indicates 
what  material  can  be  obtained  and  what  request  forms  you  may  use  to  obtain  it. 

In  the  ACM  Repository  there  are  approximately  1500  documents  of  many 
types:  Journals,  articles,  manuals,  mimeographed  reports,  and  others,  on  the 
subject  of  computing  machinery.  Each  of  these  documents,  distinguishable  from 
every  other  by  even  so  little  as  a  different  color  of  cover,  has  been  given  an 
identifying  accession  number.  Every  document  has  been  described  as  completely 
as  possible  by  its  bibliographic  elements  (Group  A),  the  topics  with  which  it 
deals  (Group  B),  and  the  values  assinged  to  certain  of  its  attributes  (Group  C). 
Some  or  all  of  this  descriptive  material  is  available  for  every  document  vhich 
has  been  entered  in  the  system  and  which,  therefore,  has  an  accession  number. 

The  information  categories  describing  a  document  are  as  follows: 


20  - 


Group  A  -  A1  Authors 

A2  Date  of  publication 
A3  Title 

Ah  Editors  or  contributors 
A5  Publisher  or  issuing  agency 
A6  Physical,  dimensions  in  cm. 

A?  Number  of  pages 
Ad  Number  of  illustrations 
Group  B  ----  Index  terms 

Group  C  — -  Added  Information  (See  Partial  Key  to  Code  Used 
for  "Added  Info"  for  the  A6m  Repository) 

The  information  retrieval  system  that  you  vill  be  using  will  refer  you  to 
those  documents  which  appear  to  be  related  in  some  way  to  your  problem.  If,  for 
example,  you  request  information  about  "programming,"  the  system  will  retrieve 
all  documents  which  have  been  indexed  under  that  term.  Similarly,  you  may  aak 
for  all  articles  by  a  certain  author,  with  certain  words  in  their  titles,  from 
a  particular  source,  produced  during  a  certain  period,  etc. 

At  the  present  time  the  system  will  retrieve  material  in  answer  to  three 
forms  of  logical  requests;  "and,"  "and  not,"  "or".  That  is,  you  may  aak  for 
all  documents  written  by  John  Doe  and  on  disk  files  (conjunction);  you  may  ask 
for  all  documents  written  by  John  Doe  and  not  about  disk  files  (negation);  or 
you  may  ask  for  all  documents  written  by  John  Doe  or  about  disk  files  (disjunc¬ 
tion).  You  say  also  use  combinations  of  these  logical  expressions.  An  example 
might  be:  documents  (by  John  Doe  or  George  Smith)  and  (on  peripheral  storage 
and  not  disk  files)  and  (written  in  I960  or  1961  or  1962)  and  not  (published 
by  IBM).  After  a  retrieval  has  been  executed,  you  will  be  informed  how  many 
documents  have  been  "retrieved."  You  may  then  request  the  print-out  of  any  or 
all  of  the  information  categories  describing  these  documents. 

In  addition  to  these  logical  request  forms,  you  may  use  &  "combinatorial" 
request,  involving  up  to  8  terms.  For  example,  Combine  (author)  John  Doe, 

(index  term)  disk  files,  (index  term)  peripheral  storage,  and  (publisher)  IBM. 
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In  this  case,  the  system  will  inform  you  hov  many  documents  indexed  by  k  of  the 
terms  have  been  retrieved,  hov  many  indexed  by  ?,  by  2,  end  by  1.  You  will  then 
be  giver  the  opportunity  of  seeing  a  print-out  of  information  on  documents 
included  in  any  of  the  U  sets.  Please  note:  logical  operators  (and,  and  not, 
or)  may  not  be  incorporated  in  a  combinatorial  request. 

It  should  be  noted  that  since  the  quality  of  indexing  lias  not  always 
been  vhat  it  should  be  and  since  the  system  does  not  yet  recognize  synonymic 
expressions,  you  will  be  well  advised  to  try  different  methods  of  requesting 
information  whenever  the  results  of  a  search  are  not  pleasing  to  you.  If  you 
can  think  of  more  than  one  term  for  the  subject  under  search  (programming, 
coding,  language),  be  sure  to  use  all  possibilities. 

As  we  have  stated  earlier,  suppose,  for  the  purposes  of  this  experiment, 
that  you  are  writing  a  paper  on  ’’Artificial  Languages,"  including  algebraic 
languages,  business-oriented  languages,  list  processing,  etc.  The  emphasis  of 
your  search  is  to  be  on  the  history  of  the  development  of  automatic  programming. 

The  goal  of  your  search  is  a  list  of  accession  numbers  for  documents 
which  jfou  think  will  be  of  value  in  writing  this  paper.  Attempt  to  make  this 
list  as  complete  as  possible j  at  the  same  time,  on  the  basis  of  what  you  can 
determine  from  the  document  descriptions,  try  to  make  your  final  list  as  precise 
as  you  can  (that  is,  as  pertinent  to  the  problem  as  possible). 

You  will  be  allowed  20  minutes  in  which  to  communicate  with  the  system 
through  the  teletypewriter  operated  b;  "Shelly  Newman.  You  should  work  as 
quickly  as  possible  during  this  time,  but  you  should  not  necessarily  consider 
speed  as  your  aim.  The  completeness  and  precision  of  your  final  list  is  most 
important.  Although  it  is  expected  that  your  search  techniques  will  improve 
as  you  become  more  familiar  with  the  system,  you  should  realize  that  this 
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experiment  Is  being  performed  as  a  test  of  the  system  and  not  of  your  research 
abilities. 

After  you  have  completed  the  20  minutes  of  communication  vlth  the  system, 
you  will  receive  the  print-out  of  it  from  the  teletypewriter  and  will  be  allowed 
10  minutes  in  which  to  review  the  print-out  and  decide  on  your  final  list  of 
accession  numbers. 

Your  participation  in  this  experiment  is  greatly  appreciated.  !Fhank  you. 
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Apmiiiix  b 

DIRECTIONS  FOR  TEST  STUDY  II 

PURPOSE:  We  are  conducting  studies  of  a  computerised  information  retrieval 
system.  The  retrieval  system  consists  of  a  computer  and  a  teletypewriter. 
Stored  in  the  computer  is  descriptive  information  on  1500  documents,  including 
journals,  articles,  manuals,  reports,  and  lecture  notes,  on  the  subject  of 
computing  machinery  and  programming.  Using  the  teletypewriter  as  a  means  of 
communication,  a  person  can  type  a  request  for  different  kinds  of  information 
regarding  the  various  documents.  After  processing  the  request,  the  computer 
will  retrieve  the  information  from  its  files  and  then  print  the  infonnatior 
with  the  teletypewriter.  If  desired,  the  user  can  continue  to  make  requests 
and  obtain  information.  In  this  way,  a  person  can  quickly  and  comfortably 
retrieve  information  about  library  documents.  The  computer  does  the  actual 
searching  and  sorting. 

We  would  like  you  to  help  us  test  the  capabilities  and  accuracy  of 
computer  performance  by  requesting  information  from  the  system. 

THE  TASK:  Assume  that  you  are  writing  an  essay  on  the  history  and  development 
of  FORTRAN,  a  computer  programing  language.  You  are  looking  for  several 
documents  which  will  be  helpful  research  material  for  the  essay.  You  learn 
that  the  retrieval  teletypewriter  will  be  unemployed  for  half  an  hour  or  so. 
Being  eager  to  start  research,  you  can  now  use  the  available  teletypewriter 
to  find  titles  and  other  information  about  documents  describing  FORTRAN.  This 
retrieved  information  can  lead  you  directly  to  valuable  research  ioaterial 
shelved  in  the  engineering  library. 

In  order  to  be  using  the  teletypewriter-computer  communication,  you 


type  the  following: 
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1  —  In  answer  to  I  AM:-,  type  your  identification  number  as  follows: 

003333  <  > 

2  —  In  answer  to  THE  OPERATING  MODE  IS:-,  type  the  following: 

SEARCH  <  > 

3  —  In  answer  to  YOU  MAY  PROCEED:-,  begin  the  search. 

Note  that  you  must  type  <  >  to  indicate  the  end  of  your  messages.  Otherwise, 
the  computer  will  not  recognize  that  you  have  finished  typing. . 

Furthermore,  when  you  desire  additional  guidance,  the  computer  will 
provide  assistance  and  explanations. 

After  you  have  found  titles  of  three  documents  that  you  think  will  be 
helpful  for  an  essay  on  FORTRAN,  please  record  the  titles  on  a  piece  of  paper, 
and  present  it  to  the  experimenter.  We  are  interested  in  seeing  if  the  computer 
retrieves  relevant  information  and  prints  information  accurately. 

Your  participation  in  this  test  study  is  greatly  appreciated.  Thank  you. 
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